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Summary 

We  are  developing  a  theory  of  parsing  and  generation  which  will  bridge  between 
language-multimedia  surface  structure  and  internal  meaning  structures.  The  idea  is  that  a 
system  could  receive  a  variety  of  perceived  data  and  this  could  include  spoken  and/or 
typed  natural  language,  visuaJ  information,  tactile  information  and  possibly  other  inputs. 
It  will  then  use  a  new  Idnd  of  multimedia  grammar  to  parse  those  inputs  and  discover 
their  associated  structure.  Finally,  it  will  use  the  discovered  structure  to  translate  the 
input  into  meaning  structures  that  can  be  used  internally  for  understanding  an  input  and 
deciding  on  appropriate  answers. 

Our  theory  at  this  point  is  bidirectional.  Specifically,  the  same  algorithm  used  to 
parse  inputs  can  be  used  to  generate  outputs.  If  a  meaning  structure  is  to  be  expressed  to 
the  user,  the  system  must  find  the  set  of  language  constructs  that  will  express  the  target 
meaning.  This  involves  a  parsing-like  process  in  which  the  stmcture  of  the  meaning  is 
analyzed  and  the  correct  rules  needed  to  express  the  meaning  are  discovered.  Here  those 
rules  then  lead  to  a  set  of  multimedia  language  constructs  that  can  be  assembled  to 
express  the  originally  given  meaning. 

The  theory  includes  the  specification  of  a  set  of  operators  01,  02, 03, . . . ,  Orn 
each  with  a  syntactic  component,  a  semantic  component,  and  some  applicability  criteria. 
The  parsing  algorithm(which  can  also  be  used  for  generation)  searches  for  the  sequence 
of  operators  Oi  that  will  account  for  the  observed  surface  structure.  Their  semantic  parts 
then  are  invoked  to  find  the  target  meaning.  A  key  to  the  research  is  to  find  a 
representation  for  the  operators  that  will  enable  fast  and  efficient  processing. 


Our  current  work  is  to  develop  this  theoretical  model  and  to  code  a  prototype 
which  can  be  used  for  experimentation. 
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